Developing and Evaluating a Query Recommendation Feature to Assist Users with Online Information Seeking and Retrieval

Many search systems provide users with recommended queries during online information seeking. Although usage statistics are often used to recommend queries, this information is usually not displayed to the user. In this study, we investigate how the presentation of this information impacts use of query suggestions. Twenty-three subjects used an experimental search system to find documents about four topics. Eight query suggestions were provided for each topic: four were high quality queries and four were low quality queries. Fake usage information indicating how many other people used the queries was also provided. For half the queries this information was high and for the other half this information was low. Results showed that subjects could distinguish between high and low quality queries and were not influenced by the usage information. Qualitative data revealed that subjects felt favorable Kelly, Diane: Developing and Evaluating a Query Recommendation Feature to... Final report of a 2008 OCLC/ALISE Library and Information Science Research Grant project. Available online at: http://www.oclc.org/research/grants/reports/2008/kelly2008.pdf Page 20 of 24 about the suggestions, but the usage information was less important for the search task used in this study. Relation to Initial Work: We used the basic experimental infrastructure developed for the initial study in this study. This study was a follow-up to the initial study in that we examined (1) if subjects could distinguish between high and low quality query suggestions and (2) if subjects’ selections were influenced by usage information. [3] Kelly, D. & Gyllstrom, K. (2011). An examination of two delivery modes for interactive search system experiments: Remote and laboratory. Proceedings of the 29th ACM Conference on Human Factors in Computing Systems (CHI '11), Vancouver, Canada. Abstract: We compare two delivery modes for interactive search system (ISS) experiments: remote and laboratory. Our study was completed by two groups of subjects from the same population. The first group completed the study remotely and the second group completed the study in the laboratory. We compare differences in participants, participation behaviors, search behaviors and evaluation behaviors. Overall, for most measures no significant differences were found, but there were some notable differences. Greater variance was observed in time taken and number of documents opened and saved by remote subjects. Lab subjects provided more favorable responses to exit questionnaire items and reported significantly higher satisfaction. Lab subjects also provided significantly longer responses to open questions, while remote subjects provided more null responses. These results suggest that many behaviors do not change significantly according to study mode and that results from remote ISS experiments are similar to those from laboratory experiments. We compare two delivery modes for interactive search system (ISS) experiments: remote and laboratory. Our study was completed by two groups of subjects from the same population. The first group completed the study remotely and the second group completed the study in the laboratory. We compare differences in participants, participation behaviors, search behaviors and evaluation behaviors. Overall, for most measures no significant differences were found, but there were some notable differences. Greater variance was observed in time taken and number of documents opened and saved by remote subjects. Lab subjects provided more favorable responses to exit questionnaire items and reported significantly higher satisfaction. Lab subjects also provided significantly longer responses to open questions, while remote subjects provided more null responses. These results suggest that many behaviors do not change significantly according to study mode and that results from remote ISS experiments are similar to those from laboratory experiments. Relation to Initial Work: We used data collected in the initial study from the 35 subjects in the study which was conducted to obtain queries and terms to populate the user-generated suggestion system. We compared data generated by these subjects with a subset of subjects from the initial study (the subjects who used the system-generated suggestions) to see if and how study mode (remote vs. laboratory) impacted search and evaluation behaviors. [4] Niu, X. & Kelly, D. (under review). The use of query suggestions as idea tactics during information search. Submitted to the Journal of the American Society for Information Science and Technology. Abstract: Query suggestion is a common feature of many information search systems. While much research has been conducted about how to generate suggestions, fewer studies have been conducted about how people interact with and use suggestions. The purpose of this paper is to investigate how and when people integrate query suggestions into their searches and the outcome of this usage. The paper further investigates the relationships between search expertise, topic difficulty and task stage and query suggestion usage. A controlled laboratory study was conducted with 23 subjects who used an experimental search system with query suggestions to conduct four searches. Results showed that subjects integrated the suggestions into their searching fairly quickly and that subjects with less search expertise used more suggestions and saved more documents. Subjects also used more suggestions during the latter stages of search and when searching for more difficult topics. These results show that query Query suggestion is a common feature of many information search systems. While much research has been conducted about how to generate suggestions, fewer studies have been conducted about how people interact with and use suggestions. The purpose of this paper is to investigate how and when people integrate query suggestions into their searches and the outcome of this usage. The paper further investigates the relationships between search expertise, topic difficulty and task stage and query suggestion usage. A controlled laboratory study was conducted with 23 subjects who used an experimental search system with query suggestions to conduct four searches. Results showed that subjects integrated the suggestions into their searching fairly quickly and that subjects with less search expertise used more suggestions and saved more documents. Subjects also used more suggestions during the latter stages of search and when searching for more difficult topics. These results show that query Kelly, Diane: Developing and Evaluating a Query Recommendation Feature to... Final report of a 2008 OCLC/ALISE Library and Information Science Research Grant project. Available online at: http://www.oclc.org/research/grants/reports/2008/kelly2008.pdf Page 21 of 24 suggestion can provide support in situations where people have less search expertise, greater difficulty searching and at specific stages during the search. Relation to Initial Work: This work was a secondary analysis of data collected in [2] above. In both the initial study [1] and study [2], subjects told us during interviews and stimulated recall that the query suggestions gave them ideas for searching and helped them get started with their searches. They further stated that the suggestions helped them think of other avenues for searching when they exhausted their own ideas and when they were uncertain about how to proceed. This study was conducted to better understand when and how these subjects used query suggestions. The study also examined contextual factors (search experience, topic difficulty and search stage) that might impact usage of query suggestions. [5] Bailey, E. W., Kelly, D., & Gyllstrom, K. (2009). Undergraduates' evaluations of assigned search topics. Proceedings of the 32th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '09), Boston, Massachusetts, 812-813. [Poster] Abstract: This paper evaluates undergraduate students’ knowledge, interests and experiences with 20 topics from the TREC Robust Track collection. The goal is to characterize these topics along several dimensions to help researchers make more informed decisions about which topics are most appropriate to use in experimental IIR evaluations with undergraduate student subjects. This paper evaluates undergraduate students’ knowledge, interests and experiences with 20 topics from the TREC Robust Track collection. The goal is to characterize these topics along several dimensions to help researchers make more informed decisions about which topics are most appropriate to use in experimental IIR evaluations with undergraduate student subjects. Relation to Initial Work: This work is a secondary analysis of the data collected during the initial study. In the initial study, subjects evaluated the search topics along several dimensions including interest, personal relevance, familiarity and difficulty. The major objective of this work was to provide a characterization of these topics so that researchers could have more information when trying to select which topics to use from the collection in their studies. [6] Mack, Adam (2011). The Power of Suggestion: An Evaluation of the Positive Effects of Source and Position on the Use of Query Suggestions. Completed in partial fulfillment of the MS in Information Science, School of Information and Library Science, University of North Carolina, Chapel Hill, NC. Abstract: This study examines whether the perceived source of query suggestions and their location within the search interface affects users’ likelihood of clicking on the suggested queries. Participants were asked to complete three search tasks using the search system provided. The query suggestions were placed on either left or right side of the page, and were labeled as either system-generated or as having come from other users of the system. Participants also evaluated their engagement with the search tasks and the quality and usefulness of the query suggestions. Results indicated that users presented with query suggestions on the left scored significantly higher on two measures of engagement. While the effects of source did not meet tests of statistical significance, participants who believed the query suggestions came from other users had higher mean scores for three of the four search engagement scales and the query suggestion rating scale. This study examines whether the perceived source of query suggestions and their location within the search interface affects users’ likelihood of clicking on the suggested queries. Participants were asked to complete three search tasks using the search system provided. The query suggestions were placed on either left or right side of the page, and were labeled as either system-generated or as having come from other users of the system. Participants also evaluated their engagement with the search tasks and the quality and usefulness of the query suggestions. Results indicated that users presented with query suggestions on the left scored significantly higher on two measures of engagement. While the effects of source did not meet tests of statistical significance, participants who believed the query suggestions came from other users had higher mean scores for three of the four search engagement scales and the query suggestion rating scale. Kelly, Diane: Developing and Evaluating a Query Recommendation Feature to... Final report of a 2008 OCLC/ALISE Library and Information Science Research Grant project. Available online at: http://www.oclc.org/research/grants/reports/2008/kelly2008.pdf Page 22 of 24 Relation to Initial Work: This study used the research infrastructure developed during the initial study as well as the queries generated by subjects. It extended the initial study, by investigating whether the location of the query suggestions impacted use and whether the explicitly stated source of the suggestion (computer or human) impacted use. 10 Summary of Student Involvement Six students have benefited from this project. Karl Gyllstrom, UNC Computer Science Ph.D. student, and Earl Bailey, UNC SILS Ph.D. student, were involved with the initial study. Karl was paid with the grant funds and Earl was funded through the SILS. Karl was involved with all aspects of the project and in particular, developing the system, query suggestion features and other research infrastructure. Karl was involved with the entire life-cycle of the project (January 2008-December 2009). Earl joined the team in Fall 2008 and primarily helped with administering the study to subjects and data analysis. The infrastructure developed in this project was later used in two other studies. One study was a collaborative effort between the PI and three students (two SILS Ph.D. and one Master’s student) [2] and another was a Master’s student project which the PI advised [6]. Although no longer being paid and having finished his Ph.D., Karl Gyllstrom continued to be involved with all of these projects primarily through the provision of technical assistance.